-
Notifications
You must be signed in to change notification settings - Fork 993
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix the scheduler panic whenever the GPU is lost on node #306
Conversation
Welcome @william-wang! |
Travis tests have failedHey @william-wang, TravisBuddy Request Identifier: 3cb72fa0-a173-11e9-b4a0-0f4450928f2b |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@william-wang Could you add more comments why this can cause a panic?
ni.Releasing.Add(ti.Resreq) | ||
ni.Idle.Sub(ti.Resreq) | ||
case Pipelined: | ||
ni.Releasing.Sub(ti.Resreq) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this face the same problem?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, So I deleted this line and replace it to line 191-193.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@william-wang Could you add more comments why this can cause a panic?
In customer's environment, when the GPU was lost. the ni.Idle decreased and the value was less than ti.Resreq. Then the panic happened in Sub func. So we need to add a judgement before the execution of ni.Idle.Sub(ti.Resreq).
/lgtm |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: k82cn, william-wang The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
@asifdxtreme , please help to cherry-pick this one to kube-batch :) |
Enable LeaderElect for both scheduler and controller
@william-wang Can you please take out some time and sign cncf-cla https://github.com/kubernetes/community/blob/master/CLA.md#the-contributor-license-agreement , this PR's needs to be cherry picked to kube-batch and kube-batch has mandatory check for cncf-cla |
@asifdxtreme , I think we also need cncf-cla check for volcano :) |
okay we will add it |
For CNCF Cla check we need webhook secret from CNCF, we contacted the admins to get the secret but they replied back saying they cannot enable webhook for external repo's as of now. |
What this PR does / why we need it:
The volcano-sh/scheduler panics when startup whever the GPU is lost on nodes.
The cause is that the GPU resource descreaced, the node.idle <task.Req. The panic happened
when ni.Idle.Sub(ti.Resreq). This PR is used to fix the issue.
Which issue(s) this PR fixes (optional, in fixes #(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #294
Special notes for your reviewer: